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HARMONIZABLE,  CRAMER,  AND  KARHUNEN 
CLASSES  OF  PROCESSES 

M.  M.  Rao 

University  of  California,  Riverside,  Ca.,  92521 
1.  Introduction.  If  {Xt,t  €  T}  is  a  family  of  random  variables  with  zero 
means  and  finite  variances,  then  it  is  termed  a  second  order  centered  process  (or 
time  series).  Several  subclasses  of  such  processes  and  their  analyses  are  discussed 
in  this  article.  They  are  taken  to  be  centered  for  convenience.  The  simplest  and 
most  well-understood  class  is  the  stationary  one.  This  is  a  second  order  process 
whose  covariance  r  is  a  continuous  function  which  is  invariant  under  shifts  of  the 
time  a-ria  T  taken  to  be  either  integers,  T  =  Z  (the  discrete  case),  or  the  real  line, 
T  =  R  (the  continous  case).  Thus  in  either  case  one  has: 

r(«,t)  =  E(X.Xt)  =  [  X.XtdP ,  a,t,eT  (1) 

J  o 

and  r(s,  t)  =  r(s  -  t)  which  depends  on  the  difference  of  s  and  t.  Writing  r  for  f, 
it  follows  from  the  classical  theorems  of  Herglotz  and  Bochner  that  r  is  expressible 
as: 


«•<— (2) 

where  T  —  [0,  2jt)  if  T  —  Z  and  t  =  R  if  T  =  R.  Here  F  is  a  nonnegative, 

A 

nondecreasing  bounded  function  on  T,  called  the  spectral  function  of  the  process. 
The  importance  of  the  stationary  class  in  electrical  and  communication  engineer¬ 
ing  is  well-known  and  a  good  expositon  may  be  found  in  Yaglom  [34]  where  the 
pioneering  works  of  Wiener  and  Kolmogorov  are  also  discussed.  Many  statistical 
problems  on  these  processes  have  been  treated  by  Grenander  and  Rosenblatt  [11], 
and  by  Parzen  [22]  who  includes  some  related  extensions. 

In  a  number  of  other  applications,  stationarity  is  an  unacceptable  restriction. 

» 

Since  one  is  not  certain  about  the  fulfillment  of  the  stationarity  assumption,  it  is 
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at  least  desirable  to  have  a  knowledge  as  to  how  far  the  results  obtained  under 

this  condition  are  still  valid  when  it  is  relaxed.  In  other  words,  one  wants  to  know 

whether  there  is  some  kind  of  “robustness”  for  this  work.  In  a  response  to  such 

questions  and  also  to  take  into  account  some  honestly  nonstationary  processes, 

» 

the  classes  of  harmonizable  and  Karhunen  families  have  been  introduced  indepen¬ 
dently  and  simultaneously  by  Loeve  [16]  and  Karhunen  [14].  Only  slightly  later 
a  common  generalization  of  both  these  classes  was  formulated  by  Cramer  [6].  An 
analysis  and  interrelations  together  with  some  of  their  extensions  constitute  the 
main  theme  of  the  present  exposition.  It  turns  out  that  harmonizable  processes, 
properly  generalized,  have  the  “robustness  properties*  alluded  to  above.  Also  their 
study  proceeds  through  Karhunen  classes  since  it  is  shown  that  the  harmonizable 
family  is  an  important  subset  of  Karhunen  processes.  A  discussion  of  Cramer’s 
class  is  included  since  technically  illuminates  the  structure  of  the  above  two 
families  and  has  independent  interest. 

There  are  two  other  important  classes  of  nonstationary  processes  that  have 
been  considered  in  recent  studies  on  the  subject.  One  is  the  class(KF),  studied 
extensively  by  Kampl  de  Feriet  and  Frenkiel  [13]  and  independently  by  Parzen 
[21]  who  termed  it  “asymptotically  stationary*,  and  by  Rozanov  [31].  The  second 
one  is  the  Cram4r-Hida  class  which  is  based  on  the  “multiplicity  theory* ,  having  . 
been  motivated  by  the  classical  work  of  Hellinger  and  Hahn  on  infinite  matrices. 

It  turns  out  that  a  stationary  process  has  multiplicity  one,  while  there  is  a  har¬ 
monizable  process  of  any  given  multiplicity,  1  <  N  <  oo,  (cf.[7],[12],(5]).  Even  in 
the  multiplicity  one  case,  there  are  several  types  of  nonstationary  processes  useful 
in  prediction  and  filtering  problems.  This  indicates  that  classes  of  nonstationary 
processes  can  be  studied,  using  different  techniques  frequently  in  the  time  domain. 

It  will  also  be  found  that  generalizations  of  spectral  ideas  have  a  role  to  play  in 
this  work.  Let  us  thus  turn  to  a  detailed  description  of  these  statements. 
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2.  Harmonziable  processes.  From  the  point  of  view  of  applications,  one  of  the 
most  useful  extensions  of  stationarity  is  harmonizability.  Since  for  nonstationary 
processes  (of  second  order)  the  covariance  function  r,  given  by  (1),  must  depend 
on  both  variables  s,  t,  it  is  natural  to  consider  those  processes  for  which  the  analog 
of  (2)  still  holds.  This  leads  to  the  following  concept,  introduced  by  Lo&ve  [16], 
and  it  is  called  strongly  harmonizable  hereafter.  Namely,  the  covariance  r  admits 
the  representation: 

r{s,  t)  =  f  f  exp{isX  -  itV)F(iA,  dA'),  s,teT,  (3) 

Jt  Jt 

where  F  is  a  covariance  function  of  bounded  variation  on  T  x  f. 

In  contrast  to  the  Bochner-Herglotz  theorem,  there  is  no  usable  characteriza¬ 
tion  of  such  an  r.  But  it  is  easily  seen  that  strongly  harmonizable  processes  exist 
in  abundance.  Indeed,  let  /  be  any  (Lebesgue)integrable  scalar  function  on  the 
line,  and  denote  by  /  its  Fourier  transform,  i.e., 

/(<)  -  /  e**/(A)<U,  t  €  R.  (4) 

J  R 

If  £  is  a  random  variable  with  mean  zero  and  unit  variance,  consider  (ACt  = 

A  _ 

£/(t),  t  €  R}.  Then  Xt  has  mean  zero  and  the  covariance  r  is  given  by 


r{s,  t )  =  E(X.Xt)  =  /(*)/(*)  =  /  /  exp[xs\  -  itX')  f{X)f{X')dXdX‘ . 

J  R  J  R 


Taking 


F(A,V) 


one  verifies  that  F  is  positive  definite  and  of  bounded  variation  so  that  the  Xt 
process  is  strongly  harmonizable.  All  finite  linear  combinations  of  such  “simple” 
processes  constitute  a  large  class  and  if  F  in  (2)  is  absolutely  continuous,  then 
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these  simple  processes  are  even  “linearly  dense”  in  a  certain  well-defined  sense. 
On  the  other  hand,  if  F  of  (3)  concentrates  on  the  diagonal  A  =  A',  then  clearly 
(3)  reduces  to  (2).  Thus  stationary  processes  are  properly  extended. 

Even  though  the  Loeve  extension  of  stationarity  is  useful,  it  does  not  go  far 
enough  to  afford  a  flexibility  for  linear  operations  on  these  processes.  Since  the  lat¬ 
ter  operations  appear  naturally  in  filtering  problems,  a  further  extension  is  needed 
to  take  care  of  these  applications.  First  let  us  consider  an  example  to  understand 
how  certain  simple  linear  operations  lead  from  stationarity  to  problems  beyond 
the  strongly  harmonizable  class.  Thus  let  Lq(P)  be  the  space  of  scalar  random 
variables  with  zero  means  and  finite  variances  so  that  a  second  order  process  con¬ 
sidered  in  this  paper  is  a  subset  of  Lq(P).  Let  the  metric  (or  norm)  be  denoted 
by  ll/H  =  Vtf(\f\2)},  /  €  Lq (P).  If  A  is  a  bounded  linear  transformation  on 
Lq(P),  so  that  AXt  =  Yt  €  Lq(P),  consider  a  stationary  (or  strongly  harmoniz¬ 
able)  process  {Xt,t  ST},  and  the  transformed  process  {Yt,t  €  T}.  If  the  range 
of  A  is  finite  dimensional,  then  the  Yt  process  is  strongly  harmonizable  (can  be 
nonstationary)  and  if  the  range  of  A  is  infinite  dimensional  then  the  Yt  process 
need  not  be  strongly  harmonizable.  For  instance,  let  T  =  Z,  Xn  =  fn,  an  or¬ 
thonormal  sequence  (hence  stationary)  in  Lq(P),  and  A  be  the  projection  such 
that  Yn  =  AXn  =  fn  for  n  >  0,  =  0  for  n  <  0.  Then  the  Yn  -sequence  is  a 
truncation  of  the  original  orthonormal  sequence,  but  it  is  not  strongly  harmoniz¬ 
able.  This  fact  as  well  as  the  preceding  general  statement  on  the  range  of  A  is  not 
entirely  simple.  The  details  maybe  found  in  [28].  Since,  as  remarked  earlier,  linear 
operations  are  important  for  (filtering  and  other)  practical  problems,  one  should 
have  an  extension  to  include  at  least  these  questions.  Fortunately  this  is  possible 
audit  can  be  formulated  as  follows. 

Let  {.Xt,t  6  T}  C  Lq(P)  be  a  process  with  r  as  its  covariance.  If  r  admits 
a  representation  of  the  form  (3)  with  F(-,  •)  as  a  covariance  function  which  is 
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not  necessarily  of  bounded  variation  (as  demanded  in  (3))  but  satisfies  only  the 
weaker  condition  of  finite  Frechet  variation,  then  the  process  is  termed  weakly 
harmonizable.  Recall  that  F  has  finite  Frechet  variation  on  R3  if 


|i*|(R  x  R)  =  <jup{ |  a‘aJ  /  f  *VMV)|  :  |a,f  <  1,  a,-  €  C 

{Ij)i  disjoint  intervals  in  R,  n  >  l}  <  oo. 

In  case  a, ay  Jr  /f  F(dX ,  dX')  is  replaced  in  the  above  by  a,-  s  1  and  Jr  \F(dX,  dA')|, 
then  one  has  the  usual  ( Vitali)  bounded  variation.  This  small  alteration  makes  an 
essential  difference  since  |.F|(R  x  R)  <  oo  can  hold  when  the  usual  variation  on  R 
is  infinite.  A  simple  example  exhibiting  this  phenomenon  is  the  truncated  series  of 
the  preceding  paragraph  whose  verification,  however,  needs  some  work.  Thus  each 
strongly  harmonizable  process  is  weakly  harmonizable  but  not  conversely,  and  the 
latter  is  a  strictly  larger  class.  But  one  has  to  make  concession  to  a  technical  prob¬ 
lem  in  this  generalization.  The  integral  in  (3)  is  in  the  usual  Lebesgue-Stieltjes 
sense  when  F  is  of  bounded  variation,  but  in  the  general  case  it  must  be  defined  in 
the  sense  of  Morse  and  Transue  [18].  The  thus  defined  MT-integral  is  somewhat 
weaker  than  the  usual  LS-one  in  that  the  generalization  does  not  admit  the  Jordan 
type  decomposition,  and  the  standard  Fubini  type  theorem  is  not  valid.  However, 
enough  usable  properties  are  available  to  proceed  with  a  substantial  amount  of  the 
work  for  many  applications.  A  systematic  account  on  the  structural  properties  of 
these  extended  processes  may  be  found  in  [28],  (cf.  also  [20]).  As  a  consequence  one 
deduces  that  if  {Xt,t  €  T}  is  weakly  harmonizable  and  A  is  a  bounded  linear  (and 
even  some  unbounded  ones  such  as  the  differential)  transformation,  Yt  =  AXt, 
then  {Ft,  t  €  T}  is  also  weakly  harmonizable.  Thus  the  latter  class  is  closed  under 
such  mappings.  For  these  reasons,  the  positive  definite  function  F  in  (3),  of  finite 
Frechet  variation,  is  often  called  the  (generalized)  spectral  function  of  the  harmo- 
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nizabie  process,  even  though  it  can  be  complex  valued.  Such  spectra  have  also 
important  roles  to  play  in  applications  such  as  sampling  the  process,  filtering  and 
even  prediction  problems. 

The  above  definition  may  be  given  a  different  (but  equivalent)  form  due  to 
Bochner  [2].  For  a  reference  it  is  stated  as  follows: 

Theorem  2.1  A  second  order  process  { Xt,t  6  T}  C  Lq(P)  is  weakly  harmo- 
nizable  iff  (=  if  and  only  if) 

(s)£(P^|s)  <  M  <  oo,  t  6  T,  for  a  constant  M  >  0, 

(«)  the  covariance  r(-,  •)  is  continuous  on  T  x  T 

(iii)sup{E(\  ST  f{t)X{t)dt\2)  :  ||/||*  <  1}  <  oo  (5) 

where  f  is  the  Fourier  transform,  given  by  (4),  for  each  integrable  f  on  T  and  the 
integral  in  (5)  is  defined  in  a  standard  manner  as  a  vector  (or  Bochner )  integral, 
H/ll.  being  the  uniform  (=  supremum)  norm  of  f. 

Even  though  both  these  harmonizability  concepts  are  generalizations  of  sta- 
tionarity,  there  is  a  deep  reciprocal  relationship  between  them.  This  is  quite  im¬ 
portant  for  some  applications.  The  following  example  gives  an  indication  of  this 
property  and  explains  the  underlying  reasons  more  vividly. 

Example.  Let  (5(t),  t  >  0}  be  a  family  of  bounded  linear  mappings  on  Lq(P ) 
such  that 

(i)  5(tx  +  v)  =  S(u)S(v),  u,  v  >  0,  5(0)  =  identity 

(ii)  l|S(«)/||  <  ll/ll,  feI%{P)  Where  ||/||2  =  E(|/|2),  and 

(iii)  l|S(«)/  -  /|(  —  0  as  u  -*■  0+. 

Such  a  family  is  usually  called  a  continuous  contraction  semigroup  on  Lq(P). 
For  any  given  Xo  6  Lq(P)  define  the  process  {Yi,  t  6  T}  as: 

Y(t)  =  S(t)X o,  if  t>0,  =  S*(-t)X o,  if  t  <  0, 
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where  S*(u)  stands  for  the  adjoint  of  S(u)  so  that  it  is  a  linear  mapping  satisfying 
the  relation 


*««(»)/)»)  =  all  f,i  in  Ll(P). 

Then  the  Y(t)  -process  can  be  shown  to  be  weakly  harmonizable.  This  is  not 
obvious.  One  shows  that,  on  letting  S(-u)  =  S*(u),u  >  0,  the  family  {S(u),u  6 
T}  is  positive  definite  in  the  sense  that 

££e((s(U(  -  u ,)/„)£)  >  o,  u  s  Ll(P), 

*= i  i=i 

for  each  finite  set  {«!,.. .,un}  C  T.  This  is  easy  if  T  =  Z  and  the  case  that 
T  =  R  is  then  reducible  to  the  former.  Then  one  applies  a  form  of  the  next 
result  to  deduce  that  there  is  a  family  of  unitary  transformations  Vt  (meaning 
VtVtm  =  V*Vt  =  identity,  (i)  and  (iii)  hold),  on  a  larger  space  D  Lg(P)  such 

that  S(t)  =  QVt,t  €  T.  Here  Q  is  the  orthogonal  projection  of  L^P1)  onto  Ljj(P). 
It  should  be  noted  that  if  S{t)  =  Vt,  so  that  L% (P')  =  Ljj(P)  and  Q  =  identity, 
then  yt  =  VtXo,t  6  T,  gives  the  classical  representation  of  a  stationary  process. 
Thus  the  connection  between  these  two  clones  obtained  by  an  enlargement  of  the 
underlying  probability  space  is  an  important  and  a  deep  result. 

The  precise  statement  alluded  to  above  is  the  following: 

Theorem  2.2  Let  { Yt,t  €  T}  C  Lq{P)  be  a  given  ( weakly  or  strongly)  harmo¬ 
nizable  process.  Then  there  exists  a  possibly  enlarged  probability  space  on  which  is 
defined  Lq(P)  containing  £q(P),  an  orthogonal  projection  Q  on  Lq(P)  with  range 
Ll{P),  and  a  stationary  process  {Xt)t  €  T}(c  Lq(P))  such  that 
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Yt  =  QXt,t  €  T.  ( This  Yt  process  is  termed  a  dilation  of  the  harmonizable  Xt— 
process.)  In  the  opposite  direction,  each  stationary  process  {Xt,t  €  T}  C  Lq(P) 
and  each  continuous  linear  transformation  A  on  Lq(P)  define  [Yt  =  AXt,t  6  T}, 
as  a  weakly  (but  usually  not  strongly )  harmonizable  process  in  Lq(P). 

The  super  space  Lq(P)  is  not  generally  unique,  but  one  can  find  a  minimal 
space  with  the  desired  properties.  The  result  and  its  extended  space  may  be 
obtained  essentially  “constructively.”  It  is  related  to  some  work  of  M.A.  Naimark, 
B.  Sz-Nagy  and  others  on  Hilbert  space  operator  theory.  A  detailed  proof  with 
related  references  is  given  in  [28].  Based  on  this  result  one  can  show  that  each 
weakly  harmonizable  process  may  be  represented  in  terms  of  a  (continuous)  positive 
definite  semi-group,  as  described  in  the  preceding  example. 

The  above  theorem  enables  some  extensions  of  the  well-known  results  from  the 
stationary  theory  to  the  harmonizable  case.  For  instance,  the  following  inversion 
formula  for  F{-,  •)  of  (3)  can  be  obtained  from  the  classical  work  at  once. 

Proposition  2.3  Let  r  be  a  weakly  harmonizable  covariance  function  with  F 
as  its  representing  function-the  spectral  measure.  If  A  =  (<11,02),  B  =  (61,  b?)  are 
two  intervals  such  that  ai,bi,i  =  1,2,  are  continuity  points  of  F,  then  one  has 


F(A,B) 


=  lim  r  f 

a— oo  /  / 

00  J ~a 


—  I  S 


it 


-r(s,  t)dsdt. 


(6) 


The  generally  complex  valued  spectral  function  F  of  the  process  plays  a  cole, 
in  analyzing  harmonizable  processes  somewhat  similar  to  the  one  given  by  the 
classical  case  (2).  So  it  is  desirable  to  estimate  F  and  investigate  the  asymptotic 
properties  of  such  estimators.  This  problem,  even  in  the  strongly  harmonizable 
case,  is  not  yet  solved.  Other  unresolved  points  will  be  recorded  for  future  work, 
as  the  exposition  proceeds. 


8 


The  strongly  harmonizable  case  admits  an  extension  in  a  slightly  different 
direction.  The  covariance  function  r  of  (3)  may  be  written  as: 

r{,-t)=LL  g(s,X)g(t,V)F(dX,dX),  s,teT.  (7) 

where  g(t,\)  =  exp(itX),  which  is  2x  periodic  in  t  for  each  A  €  T.  Also  g  is 
bounded  and  jointly  continuous  in  the  variables  t,  A.  The  result  (7)  is  meaningful 
if  g  :  T  x  D  — *  C(D  C  T)  is  almost  periodic  on  T  for  each  compact  subset  D  of 
T.  More  explicitly,  a  complex  valued  continuous  function  g  on  T  x  D  is  almost 
periodic  on  T  uniformly  relative  to  D  if  for  each  compact  subset  K  of  D,  and  eac1 
e  >  0,  there  is  a  number  Iq  =  lo(e,  K)  such  that  each  interval  I  C  T  of  length 
contains  a  number  r  (called  an  £  -translation  number  of  g)  for  which  one  has 

l#(*  +  r,A)-*M)|£«,  t<ZT,\€K.  (8) 

If  D  is  a  single  point  then  g  is  called  the  classical  almost  periodic  function;  and 
in  any  case,  for  each  A  €  K  C  D,g{-,  A)  is  bounded.  With  this  concept,  a  second 
order  process  {Xt,  t  €T}  C  Lq(P)  is  termed  almost  harmonizable  if  its  covariance 
r  admits  the  representation  (7)  with  respect  to  a  family  {y(-,A),A  ET}  of  almost 

A 

periodic  functions  on  T  uniformly  relative  to  T,  and  a  covariance  function  F  of 
bounded  variation.  It  will  be  seen  in  Section  6  below  that  this  family  inherits  an 
important  structural  property  of  strongly  harmonizable  processes  of  which  it  is  an 
extension. 

From  an  applications!  point  of  view,  however,  one  should  consider  multivariate 
processes.  Thus  if  Xt  :  fl  -*  C",t  €  T,  so  that  Xt  =  (AC/, ..., ACJ*),  let  XJ  6 
Ll(P),i  —  1,  ...,n;t  6  T.  Then  the  Xt  -process  is  termed  multivariate  strongly  or 
weakly  or  almost  harmonizable  (relative  to  a  fixed  scalar  almost  periodic  g  -family 
in  the  last  case)  if  for  each  vector  a  =  (a1, ...,  an)  6  Cn,  the  scalar  process 


=  teT, 

1=1 

is  of  the  same  type  as  defined  in  the  preceding  paragraphs.  From  this  definition,  it 
follows  after  an  easy  algebraic  manipulation  that  for  each  1  <  j,  k  <  n,  and  s,  t  in 
T,  the  (cross-)  covariance  function  rjk  of  the  component  processes  X[ ,  Xf ,  is  also 


harmonizable  and  that 


r,*(i,t)  =  f  l  e“x-M'F,k(d\,dX'),  ST,  (9) 

Jf  Jt 

where  Fjk(A  x  B)  =  Fkj(B  x  A),  and  each  Fjk  is  of  respectively  usual  (Vitali) 
or  Frechet  variation  finite,  Fjj  being  positive  definite  (Fjk  need  not  be).  If  r  = 
( ryfc ,  1  <  j,k  <  n)  and  F  =  ( Fjk ,  1  <  j,k  <  n)  are  n-by-n  matrices,  then  the 
matrix  covariance  function  r  of  Xt  admits  the  representation 


r(M)=  /  fg(*,X)t(t,X')F(dX,d\'),  (10) 

Jt  Jt 

with  g(a,  A)  =  et,x  in  the  (weak  or  strong)  harmonizable  case.  The  integrals  here 
are  defined  componentwise.  Again  F  will  be  called  the  spectral  matrix  function  of 
the  vector  process  {Xt,t  €  T}. 

In  all  the  above  cases  F  has  the  following  important  property  inherited  from 

r: 

0  <  trace{j *  j '  f{X)F{dX,  <  oo  (11) 

for  any  m-by-n  matrix  function  /  with  bounded  Borel  entries.  Here  /*  is  the 
conjugate  transpose  of  /.  hi  the  stationary  case  one  has  F(X,X')  = 
where  6\\<  is  the  Kronecker  delta  and  G  is  a  positive  definite  hermitean  n-by-n 
matrix  function.  In  the  general  (e.g.,  harmonizable)  cases  the  latter  property  is 
no  longer  present  because  of  the  behavior  of  the  off-diagonal  entries  of  F  noted 
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earlier.  Before  considering  the  spectral  properties  of  the  multivariate  harmonizable 
processes,  it  will  be  necessary  to  discuss  another  extension  of  stationarity  due  to 
Karhunen  [14]  and  some  of  its  ramifications.  Let  us  introduce  this. 

S.  Karhunen  claea.  It  will  be  useful  to  motivate  the  concept  in  the  following 
way.  Consider  a  stationary  (scalar)  covariance  function  r.  By  (2)  it  has  a  spectral 
function  F  which  is  positive,  increasing,  and  bounded.  Suppose  that  F  admits  a 
density  /  (relative  to  the  Lebesgue  measure).  Then  (T  =  R  so  that  T  =  R  also) 


•(»,<)»  J  'i{~)Xn>.)dx 

= ft'"x'/7w(‘“xVJw)-dx 
=  J  k(a  +  u )h(t  +  t i)du,  8 ,  t  €  T, 


(12) 


where  h  is  the  Fourier  transform  of  y/J  (which  exists)  and  then  the  last  equality 
follows  by  the  Parseval  formula  (since  y/f  is  square  integrable).  Note  that  if  T  =  Z, 
then  h  is  a  polygonal  function  and  the  integral  in  (12)  reduces  to  a  (possibly  infinite) 
sum.  Thus  a  process  whose  covarinace  is  representable  by  a  formula- of  the  type 
(12)  relative  to  a  Borel  family{/i(t,  •),  t  €  T}  and  a  measure  p  (here  h(t,  ■)  =  h(t+-) 
and  dfi(X)  =  dX)  includes  the  stationary  class,  and  brings  in  considerable  flexibility. 
It  should  also  be  observed,  from  (12)  and  (2),  that  even  a  stationary  covariance 
can  have  different  representations,  and  this  remark  will  be  pertinent  later  on.  Let 
us  thus  present  the  desired  general  concept. 


Definition.  A  process  {Xt,t  €  T}  C  Lq{P)  with  covariance  r  is  said  to 
belong  to  the  Karhunen  class  if  there  is  an  auxiliary  measure  space  (5, 5,  u)  and  a 
set  of  complex  functions  {g(t,  •),  t  6  T}  C  L2(S,  S,  i/)  such  that 


r{ti,t2)  =  js9{ti,X)g{t2,X)v{dX),  U  €  T,i  =  1,2. 


(13) 
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Here  both  S,  T  can  be  general  sets  without  any  relation.  In  applications,  one 
usually  has  T  =  R,Z  (as  is  assumed  in  this  article)  and  then  S  =  T  (=R,  or[0,  2jt)), 
or  C,  or  such  others.  Also  u  can  be  a  nonfinite  measure  ( dv  =  d\ ,  the  Lebesgue 
measure  on  R  in  (12)  is  an  example). 

The  Karhunen  class  is  quite  large.  It  was  already  noted  that  stationary  pro¬ 
cesses  are  included  in  it.  From  the  forms  (4)  and  (13)  it  is  not  at  all  evident  that 
there  is  any  relationship  between  harmonizable  and  Karhunen  classes.  It  will  now 
be  shown  that  the  former  is  also  a  subset  of  the  latter.  This  fact  could  not  be 
obtained  until  the  availability  of  the  dilation  result  (Theorem  2.2).  It  also  depends 
on  another  classical  fact  (due  to  Cramer)  that  each  stationary  process  {JCt,  t  €  T} 
is  representable  as  : 


Xt=  eitXZ(dX),  te  R,  (14) 

j't 

where  Z(A)  €  X§(P)  and  E(Z[A)Z(B))  =  F(AnB),  for  any  Borel  sets  A,Bcf, 
the  measure  F  being  the  same  as  that  of  (2)  which  is  related  to  the  covariance 
function  r  of  the  Xt  process.  Such  a  Z(-)  is  called  an  orthogonally  scattered  measure 
by  Masani  [17].  With  this  set  up  one  has: 

Theorem  3.1  Each  weakly  harmonizable  process  {Xt,t  €  T}  is  also  a  Karhunen 
process  relative  to  a  finite  positive  measure  u  on  f  and  a  suitable  Borel  family  of 
functions  {/*,t  €  T}  in  L2(T,v). 

Proof  A  sketch  of  the  argument  follows  because  it  is  not  yet  available  in  the 
literature,  and  it  is  not  long.  Since  the  Xt  process  is  weakly  harmonizable.,  by 
Theorem  2.2  there  exists  a  stationary  dilation  {Yt,t  €  T}  C  Ljj(P)  on  a  larger 
probability  space  and  L2{P)  can  be  identified  as  a  subspace,  such  that  Xt  = 
QYt,t  €  T,  and  Q  is  the  orthogonal  projection  from  L2(P)  onto  Lq(P).  But  by 

(14), 
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t  e  T, 


and  Z  is  orthogonally  scattered.  Let  u(A)  =  S(jZ(A)|2).Then  v(-)  is  a  finite 
positive  measure  on  T .  But  by  a  classical  theorem  of  Kolmogorov  (see  [32],  p.  33, 
and  also  [17],  Thm.  5.10),  there  exists  an  orthogonal  projection  II  on  L2[T,  i/)  into 
itself  induced  by  <5  such  that 

Xt  =  QYt  =  Q([  eitXZ(d\))=  f  n(e‘'*(  ))(A)Z(dA),  t  €  T.  (15) 

JT  JT 

If  /(t,  A)  =  II(eit(  ))(A),A  €  t,  then  {ft, te  T}  C  L2{T,  v)  and  (15)  further  implies 

r{s,t)  =  E(X.Xt)  =  f  f(a, 

JT 

This  meams  that  r  has  the  representation  (13)  relative  to  {/(£,  -),t  €  T}  and  v ,  so 
that  the  Xt  is  of  Karhunen  class,  as  asserted. 

This  result  which  is  a  consequence  of  the  preceding  work,  exhibits  a  type  of  in¬ 
clusiveness  of  the  Karhunen  class  and  will  be  shown  below  to  have- a  deeper  impact 
on  the  structural  analysis  of  (multivariate)  hannonizable  processes.  It  is  however 
also  useful  to  note  another  property  of  this  family  regarding  the  existence  of  shift 
operators  on  a  nonstationary  subclass.  This  is  significant  since  harmonizable  pro¬ 
cesses  themselves  actually  do  not  admit  such  shifts  in  contrast  to  the  stationary 
class.  Let  us  explain  this  in  more  detail  because  it  is  not  at  all  obvious. 

Let  -pCe,t  6  T}  C  L%{P)  be  a  process.  For  each  a  €  T,  define  r,Xt  =  Xt+ , 
and,  if  possible,  extend  r,  as  a  linear  transformation  on  L2(P).  The  thus  extended 
r,  (also  denoted  by  the  same  symbol)  is  called  a  shift  operator  on  the  process.  If 
the  Xt  process  is  stationary  then  it  is  well-known  that  such  a  r,  exists,  and  in  fact 

£((r.Xu)(7X))  =  E(X^X+n)  =  r(u  -  v)  =  E(X%X,). 
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Thus  r,  preserves  the  lengths  (=  norms)  and  it  is  unitary.  However,  for  a  harmoniz- 
able  process  such  a  r,  need  not  exist.  For  instance,  consider  Yt  =  Xt  for  t  >  0,  =  0 
for  t  <  0  where  the  Xt  process  is  stationary.  Then  by  Theorem  2.2,  {Yt,t  €  T}  is 
weakly  harmonizabie,  but  if  u,  v  <  0,  and  s  €  R  3uch  that  ti  +  a>0,  t/-l-s>0,  for 
all  such  s, 


E{XnX„)  =  0  #  r(tt  -  v)  =  E(X*+,Xv+0). 

In  fact,  assuming  that  the  Xt  process  is  not  identically  zero,  one  has 

linil3  =  B(|V.|2)  =  0,  Jut  llr.nil2  =  £(ln+.|2)  =  r(0)  /  0. 

Hence  r,  cannot  be  linearly  extended.  Here  are  some  simple  (good)  sufficient 
conditions  for  a  shift  operator  r#  to  exist  and  be  continuous.  For  each  finite  set 
of  points  from  T,  and  complex  numbers  a*,...,  On,  if  Un  = 

then 


Un  =  Q=>  TgUfi  =  £  a iXti+.  =  0.  (16) 

1=1 

Equivalently 


implies 


ll^nl!2  =  a«a tj)  =  0 


«'=1  3=1 


\rM\2  =  S  +  •*  *i  +  •)  =  °- 

t=  1  ;'s  1 
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Then  r,  can  be  extended  by  linearity  (unboundedly  in  general)  onto  the  linear 
subspace  H  of  Lq(P)  generated  by  {Xt,t  £  T}.  Such  a  t,  will  also  be  bounded  if 
there  exists  a  number  c  >  0  such  that 

l|r,{/„||  <  c||tT„||.  (17) 

In  the  stationary  case  e  =  1  and  there  is  equality  in  (17).  As  an  easy  consequence  of 
(16),  one  will  have  rtr,i  =  rt+t>  and  hence  on  the  relevant  subspace  H ,  {r,,  a  €  T} 
should  form  a  semi-group.  Since  this  is  true  for  the  stationary  case  (with  r,  as 
unitary)  and  since  one  wants  to  include  some  nonstationary  processes,  it  is  natural 
to  look  for  the  r,  family,  with  some  structure,  at  least  as  a  normal  operator  semi¬ 
group,  i.e.  {r„,  a  6  T}  should  satisfy  the  commutativity  relations  r,r*  =  r*r,  (r*  is 
the  adjoint  of  r,).  Let  us  find  out  possible  nonstationary  processes  admitted  under 
such  an  assumption,  since  the  stationary  class  is  automatically  included  (because 
every  unitary  operator  is  normal).  The  mathematical  detail  will  be  minimized 
here. 

Let  {r„  a  >  0}  be  a  bounded  semi-group  of  normal  shifts  on  {ATt,  t  >  0}  such 
that  | \t,X— X 1 1  — *■  0  as  s  — *■  0  for  each  X  €  U ,  the  closed  span  of  the  Xt  in  Lq(P).  In 
order  to  include  the  unitary  (or  equivalently  the  stationary)  case,  r#  should  not  be 
assumed  self-adjoint!  Thus  normality  is  the  next  reasonable  generalization.  [Also 
the  condition  that  \\r,X—X [|  — ►  0  is  known  to  be  equivalent  to  the  strong  continuity 
of  t,  for  a  >  0  and  the  boundedness  of  r,  on  0  <  s  <  1  as  well  as  the  density  of 
U#>or#(l0  ia  This  is  thus  a  technical  hypothesis.]  Let  =  (i\  —  I)/h,  h  >  0. 
Then  Ah  is  a  bounded  normal  transformation  for  each  h.  It  is  a  consequence  of 
the  classical  theory  of  such  semi-groups  that  for  each  X  6  M  one  has 

r.X=  lim  e,A*X,  (18) 

the  limit  existing  in  the  metric  of  M,  uniformly  in  a  on  closed  intervals  [0,  a],  a  >  0. 


On  the  other  hand,  for  each  h  >  0,  Ah  is  a  bounded  normal  operator  on  the  Hilbert 
space  H.  Hence  one  can  invoke  the  standard  spectral  theorem  according  to  which 
there  existis  a  “resolution  of  the  identity” ,  {^(A),  A  C  C}  such  that 

AkX  =  J  zEh(dz)X ,  X  e  tfo,  (19) 

where  the  integral  is  a  vector  integral  and  /i£(A)  =  E^(A)X  6  H,  gives  a  vector 
measure.  Here  Ho  C  H  is  the  3ubspace  for  which  the  integral  exists,  l.e.  z  is 
integrable  for  X  €  Hq.  But  from  the  same  theory  one  can  abo  deduce  that 


e^X  m  J  e'*Eh(dz)X,  X  e  Hi  C  H0  (20) 

for  which  e§*  is  ^-integrable.  If  y*  €  H*,  then  y*E/±(  )X  is  a  signed  measure  in 
(20)  and  if  y*  *=  J£(€  Hm  =  H)  then  it  is  a  positive  bounded  measure  for  each 
h  so  that  one  can  invoke  the  Helly  selection  principle  and  then  the  Helly-Bray 
theorem  in  one  of  its  forms  to  conclude  that  lim^—o  y* Eh{')X  converges  to  some 
vs,ymi  a  signed  measure.  This  may  be  represented  as  y*F(-)X  for  an  F(  )  which 
has  properties  analogous  to  those  of  En(-).  Here  the  argument,  which  is  standard 
in  spectral  theory,  needs  much  care  and  detail.  With  this  one  can  take  limits  in 
(20)  as  h  — ►  0  and  interchange  it  with  the  integral  to  get 

TgX  =  lim  c*Al' X  —  f  e,MF(dz)X.  (21) 

a— o  Jq 

Thus  the  measure  F{  )X  is  orthogonally  scattered  and  is  supported  by  the  inter¬ 
section  of  the  spectral  sets  of  Ah,  h  >  0.  It  now  follows  that,  if  X,  =  r,X o,  then 
by  (21)  with  X  =  X0{e  Hi)  there,  one  gets 


X,  =  J  e'xZ{d\),3>0 


(22) 
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where  Z(-)  on  C  is  an  Lg(P)-  valued  orthogonally  scattered  measure.  The  covari¬ 
ance  function  r  of  this  process  is  given  by 

r(a,t)  =  E{Xt Xt)  =  f  exp(az  +  tz)  G(dz),  (23) 

J  c 

with  G(A(1  B )  =  E(Z(A)Z(B)).  If  S  =  C  and  1/  =  G  in  (13),  one  sees  that 
{X,,s  >  0}  is  a  Karhunen  process  relative  to  f(a,  -),a  >  0 ,f(a,z)  =  e**,  and  the 
finite  positive  measure  G  such  that  f(s,  ■)  E  L2(C,G),a  >0.  If  C  is  replaced  by 
its  imaginary  axis,  and  for  a  <  0  the  process  is  extended  with  X,  =  rZ.,X o,  then 
the  stationary  case  is  recovered  (cf.  (2)).  That  (23)  is  essentially  the  largest  such 
subclass  of  Karhunen  processes  admitting  shifts  again  involved  further  analysis 
and  this  was  shown  by  Getoor[9]  in  detail. 

Thus  the  Karhunen  class  contains  a  subset  of  nonstationary  processes  which 
admit  shift  operations  on  them  and  also  a  subset  of  nonstationary  processes  (namely 
the  harmonizable  class)  which  do  not  admit  such  transformations.  Since  the  repre¬ 
senting  measures  in  (3)  and  (23)  or  (13)  are  of  a  different  character  (it  is  complex 
“bimeasure”  in  (3)  and  a  regular  signed  measure  in  (13)),  a  study  of  Karhunen  pro¬ 
cesses  becomes  advantageous  for  a  structural  analysis  of  various  stochastic  models. 
On  the  other  hand  (3)  shows  a  close  relationship  of  some  processes  with  a  pos¬ 
sibility  of  employing  the  finer  Fourier  analytic  methods,  giving  perhaps  a  more 
detailed  insight  into  their  behavior.  Thus  both  of  these  viewpoints  are  pertinent 
in  understanding  many  nonstationary  phenomena. 

4.  Cramir  data.  After  seeing  the  work  of  the  preceding  two  sections  it  is 
natural  to  ask  whether  one  can  define  a  more  inclusive  nonstationary  class  incor¬ 
porating  and  extending  the  ideas  of  both  Karhunen  an  Loeve.  Indeed  the  answer 
is  yes  and  such  a  family  was  already  introduced  by  Cramer  in  1951,  [6],  and  a  brief 
description  of  it  is  in  order.  This  also  has  an  independent  methodological  interest 
since  it  results  quite  simply  under  linear  transformations  of  Karhunen  classes  in 
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much  the  same  way  that  harmonizable  families  result  under  similar  mappings  from 
the  stationary  ones. 

One  says  that  a  function  F  on  T  x  T  into  C  is  locally  of  (Frechet)  variation 
finite  if  the  restriction  of  F  to  each  finite  proper  subrectangle  I  x  I  of  T  x  T  haw 
the  (Frechet)  variation  finite,  I  C  T  being  a  finite  interval.  Let  us  now  state  the 
concept  in: 

Definition.  A  second  order  process  {-Yt,£  6  T}  C  L% (P)  is  of  Cramer  clots 
(or  class  (C))  if  its  covariance  function  r  is  representable  as: 

r(ti,t2)  =  J  J  9{*i,  X)g(t2,X')»(dX,dX'),  U  6  T,i  =  1,2,  (24) 

relative  to  a  family  { g(t ,  •),{£!}  of  Borel  functions  and  a  positive  definite  function 
u  of  locally  bounded  variation  on  5x5,  S  being  a  subset  of  t  (or  more  generally  a 
locally  compact  space)  and  each  g  satisfying  the  (Lebegue)  integrability  condition: 

0  <J  J  gMW*jv(dXM')<oo,teT.  (25) 

If  u  has  a  locally  finite  Frechet  variation  and  the  integrals  in  (24)  and  (25)  are  in 
Morse* Transue  sense,  the  corresponding  concept  is  called  the  weakly  of  class  (C). 

It  should  be  noted  that,  in  (24),  v  is  of  locally  finite  variation  means  that 
v  determines  a  regular  complex  measure  v  on  S  x  S,  which  is  locally  finite.  In 
particular,  if  5  =  R,  the  variation  measure  of  v  is  a  finite.  In  the  Frechet-variation 
case,  v  does  not  determine  such  a  measure,  but  it  is  merely  a  “C-  bimeasure”  which 
is  locally  finite.  If,  however  v  concentrates  on  the  diagonal  of  5x5,  then  (24) 
reduces  to  (12)  and  the  Karhunen  class  is  thus  included  in  class  (€)  which  in  turn 
is  included  in  weakly  of  class  (C).  Here,  if  g(t,  A)  =  e,tA,S  =  T,  then  necessarily 
v  will  be  of  (Fr4chet)  variation  finite  and  the  harmonizable  class  is  realized.  Thus 
the  following  hierarchy  of  nonstationary  classes  is  obtained: 


Stationary  C  strongly  harmonizable 
C  weakly  harmonizable 


C  Karhunen  class 
C  Cramer  class 
C  weakly  of  class(C). 

All  these  inclusions  are  proper.  One  key  feature  is  that  each  member  of  these 
classes  admits  an  integral  representation  analogous  to  that  of  (14),  though  their 
proofs  in  each  case  differ  considerably. 

Remembering  the  dilation  of  a  harmonizable  series  into  a  stationary  one  as 
given  in  Theorem  2.2,  one  might  ask  for  a  similar  result  between  the  Cramer  and 
Karhunen  classes.  The  methods  and  ideas  of  proof  of  that  result  extend  to  give 
only  the  following  somewhat  weaker  statement. 

Theorem  4.1  If  {Xt,t  €  T}  C  Lq(P)  is  a  Karhunen  process  and  A  is  a 
bounded  linear  transformation  on  Lq(P)  into  itself,  then  {yt  =  AXt,t  6  T},  is  a 
process  of  class  (C)  whenever  the  representing  measure  G  of  (13)  is  finite.  The 
converse  direction  (on  dilation)  is  not  necessarily  valid.  However,  if  in  (24)  u  is 
of  finite  (Fr4chet)  variation  and  each  g(t,-),t  €  T,  is  individually  a  bounded  Borel 
function,  then  such  a  class  (C)  proces  can  be  dilated  to  a  Karhunen  process  on  a 
larger  space  Lq(P)  containing  Lq(P). 

In  this  generaliation,  it  is  significant  that  the  full  dilation  result  does  not  ob¬ 
tain.  Only  an  interesting  subclass  extends.  Details  and  related  references  with 
further  extensions  on  the  problem  can  be  found  in  [28].  It  will  appear  in  applica¬ 
tions  (cf.  Section  8  below)  that  these  classes  arise  naturally,  especially  as  solutions 
of  linear  stochastic  differential  equations  of  filtering  and  signal  extraction  prob¬ 
lems.  Another  important  reason  for  a  study  of  Karuhunen  class  will  emerge  in 
Section  7. 
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5.  Multivariate  harmonizable  processes.  An  n-dimensicmai  harm cmizabi e  pro¬ 
cesses  Xt  =  {X},...,X?),t  e  r,  as  defined  in  Section  2  (cf.(9)  -(H)),  has  its 
covariance  matrix  r  representable  as: 

r(a,t)=  [.  f  e{uX-itX’F(d\,d\'),  s,teT, 

JfJt 

for  a  suitable  n-by-n  matrix  of  C-bimeasures  F.  A  similar  multivariate  analog  of 
Cramer  and  Karhunen  processes  can  be  given  with  a  corresponding  formula  in  the 
form  of  (10).  Note  that  F  is  also  hermitean  positive  definite  for  the  Karhunen 
class  but  not  for  the  Cramer  and  harmonizable  families.  Since  there  are  not  many 
results  available  for  other  classes,  the  more  familiar  (nonstationary)  harmonizable 
case  will  be  discussed  here.  For  this,  the  spectrum  plays  a  role  somewhat  analogous 
to  that  of  the  stationary  case,  and  its  spectral  domain  is  given  as  follows. 

In  all  the  extensions  of  stationarity  considered  above,  their  covariance  func¬ 
tions  admit  “factorizable”  kernels  for  their  integrands  (cf.  (3),  (13),  (24)).  This 
fact  translates  itself  into  integral  representations  of  their  sample  paths  as  in  (22) 
or  analogously: 


Xt  =  I  g{t,\)Z(d\),  ter,  (26) 

Js 

where  Z  is  a  measure  on  5  into  Lq(P)  which  is  orthogonally  scattered  for  the 
Karhunen  case  (just  as  for  the  stationary  processes).  It  satisfies 

E(Z(A)Z(B))  =  F(A,B), 

for  the  Cramer  and  harmonizable  processes.  Here  F  is  generally  only  a  bimeasure. 
The  symbol  in  (26)  is  a  suitable  stochastic  (or  vector  measure)  integral.  It  is  these 
representations  that  make  up  a  study  of  the  related  spectral  domain,  and  inherit 
several  properties  of  the  time  domain.  Thus  in  these  cases  their  spectral  spaces 
are  given  as: 
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£*(F)  =  {/:  S  -  °|  j  j  nX)HX')F(d\,iX')  =  (/,/),«»»*.}.  (27) 

For  the  harmonizable  case  S  =  T,  and  in  all  cases  0  <(/,/)<  oo,  because  of  the 
special  relationship  between  F  and  Z.  The  (•,  •)  jives  a  (semi-)inner  product  and 
a  (semi-)  norm:  jj/||2  =  (/,  /).  For  the  multivariate  case  one  has: 

Ca{F)  -  {/ :  S M\  J  J  f{X)F{dX,dX')f(k')  =  {frf),  <ZMh};  (28) 

and  |j/||^.  =  trace(f,f )  defines  a  semi-norm,  f*  being  the  conjugate  transpose  of 
the  matrix  function  /(€  M,  the  space  of  complex  m-by-n  matrices).  Again  5  =  T 
for  the  harmonziable  case.  In  order  to  carry  out  linear  operations  for  problems 
such  as  filtering  and  prediction,  it  is  necessary  to  know  the  structural  properties  of 
the  space  £?(F).  This  is  nontrivial  and  especially  in  the  multivariate  case  it  was 
open  for  sometime  (cf.  [15]).  The  following  key  property  which  was  needed  there, 
has  recently  been  estabished  for  the  harmonizable  case  in  [30]  and  can  be  stated 
as: 

Theorem  5.X  If  {Xt,t  €  T}  it  a  multivariate  weakly  harmonizable  process 
With  F  as  its  spectral  matrix  function  and  £2(F),  defined  by  (28),  is  its  spectral 
domain  space  then  (£2{F),  ||  •  ||p)  is  complete  in  the  sense  that  it  is  a  Hilbert  space 
of  equivalence  classes  of  matrices  with  inner  product  defined  by  ((•,  •))  =  trace(-,  •) 
where 

(/,  J)  =  f.  !  n\)F(d\,dX')g-(X'),  (29) 

Jt  Jt 

the  ustar  ”  denoting  the  conjugate  transpose.  Here  Z2[F)  is  a  linear  space  of  (com¬ 
plex)  matrix  functions  on  f  with  constant  matrix  coefficients  for  linear  combina- 


In  order  to  assert  a  similar  property  for  the  class(C)  or  Karhunen  class,  it  will 
be  necessary  to  restrict  the  family  {g(t,  •),  t  €.T}  suitably. 

The  importance  of  the  above  property  is  better  understood  if  one  looks  at  an 
application.  The  following  is  one  such  for  signal  extraction.  A  general  signal  plus 
noise  model  is  given  by 


Xt  =  St  +  Nt,  t  6  T,  (30) 

where  the  St  and  Nt  are  (stochastic)  signal  and  noise  processes  both  of  which  are 
supposed  to  be  weakly  harmonizable  so  that  the  output  process  Xt  is  also,  whenever 
the  St  and  Nt  are  uncorrelated  or  harmonizably  correlated.  For  simplicity  of 
exposition  here,  let  us  assume  that  they  are  uncorrelated.  If  Fx,Ft,Fn  are  the 
(known)  spectral  functions  of  these  processes,  let  h(X)  =  F,(X,T)  +  Fn(X,T )  and 
k(X)  =  Fm(X,T).  The  knowledge  of  these  spectral  functions  is  assumed  from  prior 
considerations.  The  problem  here  is  to  estimate  Sa  optimally,  for  any  a  €  T,  based 
on  the  output  Xt  (i.e.  on  a  realization).  Here  optimality  refers  to  the  least  squares 
(or  error  mean  square)  criterion.  A  solution  of  the  problem,  using  Theorem  5.1, 
can  be  given  as 

Theorem  5.2  Let  Xt  =  St  +  Nt,t  6  T,  be  a  harmonizable  output  of  the  ( un¬ 
correlated  harmonizable)  signal  plus  noise  model  (SO).  Let  Zx(-)  be  the  stochastic 
representing  measure  of  the  Xt  process  given  by  (14)  on  (26).  Then  the  least  squares 
optimal  estimation  Sa  of  the  signal  St  att  =  a  eT,  is  obtained  as: 

l  (*a{X)Zx{dX),  (31) 

Jf 

where  the  “ signal  characteristic ”  Ga(-)  is  an  n-by-n  matrix  function  which  is  a 
unique  solution  of  the  (matrix)  integral  equation 
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(32) 


/  G.(X)h(dX)  =  /  e^k(dX), 

Jf  Jt 

the  h(-)  and  k(-)  being  the  u marginal ”  measures  of  the  spectral  functions  of  the 
St  and  Nt  processes  defined  above.  The  variance- covariance  matrix  of  the  error 
Sa  -  Sa  is  given  by 

£.=  /  /«*(*-*'>  *•.(<**,<* A')-  /  /  G^A^dA^A'^A').  (33) 

jfjf  JfJt 

To  use  this  result  in  specific  applications,  one  has  to  compute  Fx  and  Zx. 
From  the  data  one  can  calculate  the  covariance  r,  and  then  Fx  is  obtained  by 
means  of  formula  (6).  The  stochastic  measure  Zx(-)  can  also  be  obtained  by  using 
the  dilation  theorem  (cf.  Theorem  2.2)  and  a  known  result  from  the  stationary 
theory  (cf.  [32],  pp.  26-27).  Thus  for  any  interval  A  =  [a,b)  C  f  for  which 
Z{{a})  =  0  =  Z({b}),  one  has  (limits  taken  in  mean-square  sense) 


Z{A)  = 


X„tt  -  «)  +  £  £o<m<„  if  T  =  Z 

‘—fjf—Xtdt,  if  T  =  R. 


In  the  one  dimensional  stationary  case,  if  all  spectral  functions  have  densities 
/*!  /•»  fm  then  (32)  and  (33)  reduce  to  the  well-known  results  where  the  ga  and  <r0 
can  now  be  given  explicitly  as: 


ga(X)  =  e <aV.(A)/(/.(A )  +  /n(A)),^  =  E(\S0\2)  -  \g*(X)\2 ft(\)d\. 

6.  Class (KF)  and  harmonizability.  As  noted  in  the  preceding  sections,  many 
processes  which  are  extensions  of  the  stationary  ones  with  “triangular  covariances” 
admit  integral  representations  such  as  (26).  However,  there  are  other  classes  based 
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on  the  behavior  of  covariances  at  infinity.  An  important  such  family,  motivated  by 
certain  summability  methods,  is  the  one  introduced  in  the  1950’s  by  J.  Kampe  de 
Feriet  and  F.  N.  Frenkiel,  with  a  detailed  exposition  later  in  [13].  It  will  be  called 
class( KF ) .  This  was  also  independently  given,  slightly  later,  by  Yu. A.  Rozanov  [31] 
and  E.  Parzen  [21],  the  latter  under  the  name  “asymptotic  stationarity.”  Let  us 
state  the  precise  concept  here. 

Definition.  A  process  {Xt,t  £  T}  C  Lq(P)  with  a  continuous  covariance  r  is 
of  class( KF)  if  for  each  h  6  T  the  following  limit  exists: 


f(h)  = 


lima^coj;J"kr(s,s  +  \h\)ds,  if  T  =  R 

+  i  fT  =  Z. 


(34) 


It  may  be  verified  that  each  of  the  quantities  on  the  right  of  (34)  before 
taking  the  limit  is  positive  definite  so  that,  when  the  limits  exist,  f(-)  is  also.  The 
continuity  of  r  in  the  discrete  case  is  trivial  and  in  the  case  that  T  =  R,  f  is 
measurable  even  when  it  is  not  continuous.  In  either  case,  as  a  consequence  of  the 
classical  Herglotz-Bochner-Riesz  theorem  on  a  characterization  of  such  functions 
(cf.,e.g.,[29],  Sections  4.4  and  4.5)  there  is  a  unique  positive  bounded  nondecreasing 
H(  )  such  that 


f{h)  =  j  eihxH{d\),  a  ■  a  •  (/»)  6  T,  (35) 

where  a  •  a  •  (h)  refers  to  Lebesgue  measure  when  T  =  R,  and  all  h  if  T  =  Z. 
In  analogy  with  (2),  H(-)  is  termed  an  associated  spectral  function  of  the  Xt  pro¬ 
cess.  Here  several  examples  of  processes  in  class(KF),  in  addition  to  the  obvious 
stationary  family  for  which  r(h)  =  r( j/* )),  will  be  noted. 

Every  strongly  harmonizable  process  is  in  class(KF).  This  was  first  noted  by 
Rozanov  in  [31]  and  independently  later  analyzed  in  more  detail  by  Bhagavan  [l]. 
In  fact,  somewhat  more  generally,  almost  harmonizable  processes  (cf.  Sec.  2)also 
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belong  to  class(KF),  as  shown  in  [25],  It  is  an  interesting  fact  that,  in  the  strongly 
harmonizable  case,  (35)  holds  for  all  h  6  T,  so  that  f(-)  is  continuous.  Another 
example  is  provided  by  the  process  {Xt,t  €  Z }  which  is  a  solution  of 

Xt  =  aXt— i  +  ft)  4  €  Z,  (36) 

where  et?s  are  uncorrelated  random  variables  with  zero  means  and  unit  variances, 
and  |a|  <  1.  It  is  easily  seen  that  the  limit  (34)  exists  for  this  sequence.  One  can 
also  consider  kik  order  difference  equations  with  constant  coefficients,  extending 
(36),  such  that  all  the  roots  of  the  characteristic  equations  lie  inside  the  unit  circle, 
but  the  details  will  not  be  entered  into  here.  What  about  the  weakly  harmonizable 
class?  In  fact,  this  question  was  raised  in  [31].  It  turns  out  that  neither  includes  the 
other  completely.  The  preceding  example  already  gives  a  nonharmonizable  process 
of  class(KF).  The  other  noninclusion  will  now  be  discussed  since  that  provides  a 
better  insight  into  the  structure  of  both  these  classes. 

Consider  a  weakly  harmonizable  process  {Xt,t  €  T}  with  covariance  r  and 
F  as  its  representing  bimeasure  (cf.(3)).  Suppose  that  for  this  bimeasure  the 
dominated  convergence  theorem  holds  in  the  sense  that  if  |/„|  <  g  a.e.,  fn~*f 
pointwise,  and 


<  oo  =► 


/  UmWM') 

lim  f  f  fn(X)fn~)F (<«, dX')  =  [  [  f{X]J[^)F{dX,dX')  <  co.  (37) 
Jf  JT  JfJf 


If  F  has  finite  variation,  this  is  automatic  so  that  the  strongly  harmonizable  case  is 
included.  It  also  holds  for  some  F  having  only  finite  Frechet  variation.  For  instance 

A  A 

this  holds  if  F  is  of  finite  variation  locally  but  not  on  TxT  itself.  (Such  /’a  appear 
in  the  Cramer  classes.  This  statement  appeared  in  ([28],  Thm.  8.1)  without  such 
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a  precise  qualification  )  On  the  other  hand  some  restriction  is  necessary  since 
all  weakly  harmonizable  processes  are  not  in  class(KF)  as  the  following  example 
shows:  Let  {en,  — oo  <  n  <  00}  be  a  complete  orthonormal  sequence  in  a  space 
Lq(P)  so  that  the  underlying  probability  space  is  separable.  The  sequence  is  clearly 
stationary.  Let  {an,  -00  <  n  <  00}  be  a  bounded  sequence  of  numbers  and  define 
a  mapping  A  :  en  *-*■  in  Lq(P),  and  extend  it  linearly  onto  all  of  Lq(P).  This 
is  possible  since  the  en  form  a  basis.  Thus  A  is  a  bounded  linear  operator  in  Lq(P), 
and  since  the  {e„}nez  is  stationary,  Theorem  2.2  implies  that  Xn  =  Aen,n  6  Z, 
is  weakly  harmonizable.  It  will  be  shown  now  that  for  a  suitable  set  of  an’a,  the 
Xn  sequence  is  not  in  the  class(KF),  and  this  will  give  the  noninclusion  statement 
mentioned  above. 

Let  an  =  a_„,  a<j  =  1,  and  for  k  >  0,  define 


“fc  -  +  2xdJ(*) 

n=0 

where  Cn  =  [22n,22n+l)  and  Dn  =  [22n+1, 22n+2),  the  left  closed  and  right  open 
intervals.  The  sets  Cn  and  Dn  are  disjoint,  and  for  each  k  the  series  is  finite  (only 
one  nonzero  term),  1  <  a*  <  2,  k  >  0.  Then  A  defined  with  this  set  of  a„’s  is 
clearly  bounded.  The  covariance  r(k,l)  =  0  if  k  j*  l,  and  hence 


rn(h)  =  ~  ^  l’(M  +  ^)=(iy»-la2 

n  U^fe=o°fc> 


if  MO 
if  h  =  0. 


So  limn_oo  rn(h)  =  0  for  A  #  0,  but 


(  £  _ _ _i _ 

.(D)  =  2  W**' 

l  3  3.2s"  ’ 


if  n  =  22m  -  1 
if  n  =  22m+1  -  1, 


(38) 


and  hence  limm_o©  r2*m -i(0)  =  Iimm_oo  r2am+i_1(0)  =  Thus  Iimn_oo  rn  (0) 
does  not  exist.  Consequently  {Xn,  n  €  Z}  g  class(KF).  This  example  is  due  to  H. 
Niemi  (personal  communication). 
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The  preceding  computation  suggests  an  extension  of  class(KF).  Since  by  def¬ 
inition  a  process  {Xn,n  6  Z}  C  Lq (P)  belongs  to  class(KF)  provided  that  the 
sequence  of  their  averaged  covariances  converges,  it  i3  natural  to  consider  a  wider 
class  by  looking  at  their  higher  order  averages.  Thus  one  can  say  that  a  process 
with  covariance  r  is  in  class(KF,p),  p  =  1  being  the  original  definition  (cf.(34)),  if 
the  following  limit  exists  for  each  h€Z: 

lim  r^{h)  =  r(h),  p  >  1  (39) 

n— »oo 

where 


rlrlW  =  i p  =  r ■„(*). 

The  analog  for  the  case  that  T  —  R  can  similarly  be  given.  Since  in  (34) 
rn(  )  is  positive  definite,  it  is  seen  easily  that  r£(  )  is  also  positive  definite.  Hence 
f(  )  satisfies  the  same  hypothesis  and  (35)  holds,  so  that  the  representing  H(-)  may 
now  be  called  a  pth  order  associated  spectrum.  The  classical  results  on  summability 
imply  that  if  r!1*  (A)  — *  f(h)  then  rn\h)  — »  f(h)  for  each  integer  p  >  1,  but  the 
converse  implication  is  false.  Hence  class(KF)  C  class(KF,p)  C  class(KF,p+l) 
and  the  inclusions  are  proper.  Thus  one  has  an  increasing  sequence  of  classes  of 
nonstationary  processes  each  having  an  associated  spectrum.  The  computations 
given  for  (38)  show  that  the  preceding  example  does  not  belong  even  to  the  class 
Up>i  class(KF,p).  This  also  indicates  that  weakly  harmonizable  processes  form  a 
much  larger  class  than  the  strongly  harmonizable  or.e,  and  is  not  included  in  the 
last  union. 

It  should  be  remarked  here  that  a  further  extension  of  the  preceding  class 
is  obtainable  by  considering  the  still  weaker  concept  of  Abel  summability.  The 
consequences  of  such  an  extension  are  not  yet  known,  and  perhaps  should  be 
investigated  in  future. 


The  general  idea  behind  the  classy  KF.#h -P  >  L  is  that  if  the  .given  process 
is  not  stationary,  then  some  averaging,  which  is  a  smoothing  operation,  may  give 
an  insight  into  the  structure  by  analyzing  its  associated  spectrum.  Moreover,  if 
{Xt,t  €  R}  €  class(KF),  and  /  is  any  Lebesgue  integrable  scalar  function  on  R, 
then  the  convolution  of  /  and  the  Xt  process  is  again  in  class(KF)  whenever  the 
function  <j>  defined  by  <f>(t)  =  [i?(|.X’t|2)] »  is  in  L^R)  for  some  1  <  q  <  oo.  Then 

Vt  =  (f*X)t=  f  f(t-s)X.ds ,  £  €  R,  (40) 

Jr 

where  the  integral  is  a  vector  (or  Bochner)  integral,  gives  {Yt,t  G  R}  G  class {KF). 
Thus  class(KF)  itself  is  a  large  family.  This  example  is  a  slight  extension  of  one 
indicated  in  [31]. 

7.  The  Cramdr-Hida  approach  and  multiplicity,  In  the  previous  discussion  of 
Karhunen  and  Cramer  classes,  it  was  noted  that  each  {X*,  t€T}  admits  an  integral 
representation  such  as  (26)  relative  to  a  family  {g(t,  •),£  G  T}  and  a  stochastic 
measure  Z(-)  on  the  spectral  set  5  into  Lq(P).  Both  <j(£,u)  and  Z(du)  can  be 
given  the  following  intuitive  meaning,  leading  to  another  aspect  of  the  subject. 
Thus  Xt  may  be  considered  as  the  intensity  of  an  electrical  circuit  measured  at 
time  £,  Z{du)  as  a  random  (orthogonal)  impulse  at  u,  and  g(t,  u)  as  a  response 
function  at  time  u  but  measured  at  a  later  time  £.  So  Xt  is  regarded  as  the 
accumulated  random  innovations  upto  t.  This  will  be  realistic  provided  the  effects 
are  additive  and  g{t,u)  =  0  if  u>  t.  Hence  (26)  should  be  replaced  by 

Xt=  J*  g(t,u)Z(du),  teT.  (41) 

Since  in  (26)  the  g  there  need  not  satisfy  this  condition,  that  formula  does  not 
generally  reduce  to  (41).  So  one  should  seek  conditions  on  a  subclass  of  Karhunen 
processes  admitting  a  representation  of  the  type  (41)  which  clearly  has  interesting 
applications.  Such  a  class  will  be  discussed  together  with  some  illustrations. 
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First  it  is  noted  that  each  process  {Xt,t  €  T}  C  Lq(P),  assumed  to  be  left 
continuous  with  right  limits  (i.e.  for  each  t  €  T,  E(\Xt  —  Xt_^|2)  — *  0  as  h  — ►  0+, 
and  there  is  an  Xt  such  that  E(\Xt  -ATt+kj2)  — *■  0  as  k  — *  0+,  denoted  Xt  =  Xt+o), 
can  be  decomposed  into  a  deterministic  and  a  purely  nondeterministic  part  (defined 
below).  The  deterministic  component  does  not  change  from  the  remote  past  so 
that  it  has  no  real  interest  for  further  stochastic  analysis  such  as  in  prediction 
and  filtering  problems.  Thus  only  the  second  component  has  to  be  analyzed  for 
a  possible  representation  (41).  This  was  shown  to  be  the  case  by  Cramer[7]  and 
Hida[l2]  independently,  and  it  will  be  presented  here.  ([7]  has  the  1960  reference.) 

Let  H  =  sp{Xt,t  €  T}  C  Ljj(P),  and  similarly  =  sp{X,,s  <  t}  C  M 
and  M-oo  =  ntgrl/t.  Since  Mtl  C  a  for  ti  <  t2,  one  has  H-ao  C  C  U  and 
H-ao  represents  the  remote  past  while  Ut  stands  for  the  past  and  present.  The  Xt 
process  is  deterministic  if  H-ao  —  X  and  purely  nondeterministic  if  H-oo  ~  {0}. 
Thus  the  remote  past  generally  contributes  little  to  the  experiment.  The  separation 
of  remote  past  from  the  evolving  part  is  achieved  as  follows.  A  process  {Xt,t  6  T} 
which  is  left  continuous  with  right  limits  (and  this  is  automatic  if  T  =  Z)  can  be 
uniquely  decomposed  as:  Xt  =  Yt  +  Zt,  t  €T,  where  the  Yi-component  is  purely 
nondeterministic,  the  Zt  is  deterministic  and  where  the  Yt  and  Zt  processes  are 
uncorrelated.  (This  is  a  special  case  of  Wold’s  decomposition.) 

Since  the  deterministic  part  is  uninteresting  for  the  problems  of  stochastic 
analysis,  and  can  be  separated  by  the  above  result,  one  can  ignore  it.  Hence  for  the 
rest  of  this  section  it  will  be  assumed  that  our  processes  are  purely  nondeterministic. 
The  proofs  of  the  following  assertions  may  be  completed  from  the  work  of  Cramer 
in  [7),  (cf.,  the  references  for  his  early  papers  there). 

The  approach  here  does  not  give  much  insight  if  T  =  Z.  However,  T  =  R 
is  really  the  difficult  case,  and  the  present  method  is  specifically  designed  for  it. 
The  new  element  in  this  analysis  is  the  concept  of  “multiplicity” ,  and  it  is  always 
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one  if  T  =  Z  while  it  can  be  any  integer  N  >  1  if  T  =  R.  (See  [5j,  and  the 
references  there,  and  also  [7].)  The  basic  idea  is  to  “break  up”  the  continuous 
parameter  case,  in  the  sense  that  each  such  process  can  be  expressed  as  a  direct 
sum  of  mutually  uncorrelated  components  of  the  type  (41)  so  that  each  of  the 
latter  elements  can  be  analyzed  with  special  methods.  This  relatively  deep  result 
was  obtained  independently  (cf.,  [7]  and  [12])  and  can  be  given  as  follows: 

Theorem  7.1  Let  { Xt,t  £  R}  C  Lq(P)  be  a  purely  nondeterministic  process 
which  is  left  continuous  with  right  limits  on  R.  Then  there  exists  a  unique  integer 
N,  1  <  N  <  oo,  called  the  multiplicity  of  the  process,  and  a  not  necessarily  unique 
set  of  ordered  pairs  •)>  Fjt),  1  <  k  <  N,  t  €  R}  of  the  following  description: 

(i)  gk(t,  •) :  R  -*•  C  is  a  Borel  function,  1  <  Jfe  <  N,t  €  R, 

(ii)  Ffc  :  R  — ►  R  is  a  non- decreasing  ( not  necessary  bounded)  left- continuous 
function  such  that  if  i /k(A)  =  fA  Fk{d\),A  C  11  is  a  Borel  set,  then  i/k+i  << 
i/jt,  1  <  k  <  N,  (i.e.  is  dominated  by  i/*) 

(iii)  {3fc(«,-)i®  <  *}  C  i2((— oo,4),-Kfe)  is  {-norm)  -dense;  and  if  r  is  the  covari¬ 
ance  function  of  the  Xt  process,  then 

r(*>t)  =  2/  0fc(*,A)3fc(t,A)Ffe(dA),  s,t€R,  (42) 

fc=i  J-f3° 

the  series  converging  absolutely  if  N  =  +oo. 

Using  the  Lebesgue  decomposition  of  measure  theory,  it  can  be  verified  that  if 
the  Xt  process  is  stationary  and  nondeterministic  then  in  (2)  the  spectral  function 
is  absolutely  continuous  relative  to  the  Lebesgue  measure  on  R  with  a  density  a-nH 
hence,  as  noted  in  (12),  (42)  becomes 


r(M)  =  /  g(s  -  \)g(t  -  X)d\.  (43) 

J  —  OO 

Thus  such  stationary  processes  always  have  multiplicity  unity.  The  converse  is  not 
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true.  There  exist  nonstationary  (even  strongly  harmonizable)  nondeterministic 
processes  of  multiplicty  N  for  any  given  N,  1  <  N  <  oo. 

It  should  be  noted  that  (42)  can  also  be  stated  for  the  Xt  process  using  the 
(stochastic)  integral  representations: 


*  rt 

f.  =  £/ 

i=i 


9k{t,  u)Zk(du),  t  6  R, 


(44) 


where  each  Zk(-)  is  orthogonally  scattered,  E(Zk(A)Zi(B))  =  0  if  k  l,  and 
=  Vh(A  n  B)  =  JAnB  Fk{dX)  if  k  =  l.  The  pairs  {$*(*),  •),  Fk),  1  <  Jb  <  N,  t  €  R} 
satisfy  the  previous  conditions.  Moreover,  one  has 


Ht  =  ten,  (45) 

where  Mt  was  defined  before  and  Ki,t  =  sp{Zi(-oo,  s)  :  a  <  t}  C  Lj^(P).  Also  in 
case  (fc(M)  >  0,  t  €  R,  then  writing  jfe(t,A)  =  gk[t,X)/gk(X,X),  and  Zk(dX)  = 
9t{X,  X)Zk(dX)  in  (42)  or  (44)  one  can  assume  that  gk(X,  X)  =  1,  for  convenience. 

To  get  a  better  feeling  for  this  somewhat  complicated  decomposition,  let  us 
present  a  class  of  nonstationary  processes  of  multiplicity  one. 

Theorem  T.2  Suppose  that  {Xt,t  6  R}  C  £§(P)  is  a  process  which  may 
be  represented  as  (44)  (or  (42)).  Suppose  further  that  each  gk{-,-)  satisfies  the 
following  conditions: 

(i)  9k(t,t)  =  1  (this  is  no  restriction  ifgk(t,t)  >  0),t  €  R, 

(ii)  for  each  X  <  t,gk(t,X)  and  tyf{t,  X)  exists,  continuous  and  bounded, 

(iii)  each  Fk  £  a  constant  and  has  a  (Lebesgue)  density  F'k  which  has  at  most 
a  finite  number  of  discontinuities  on  each  finite  subinterval  of  R. 

Then  the  multiplicity  of  the  process  {Xt,t  6  R}  is  one,i.e.,  N  =  1. 
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The  case  N  =  1  already  has  interesting  connections  with  other  known  classes. 
For  instance  let  g(t.  A)  =  p(t)/p( A)  in  (42)  with  N  =  1,  and  p( A)  >  Q,A  €  R.. 
Writing  /  =  F* ,  (42)  becomes 

ft\\ 

mT) V"-  (46) 

Hence  for  all  a  <  t  <  u,  if  p(s.t)  =  r(.<t,  t)/r(s,  .*),  3  <  £,  one  gets 

/>(*>«)  =  p{s,t)p{t,u).  (47) 

This  //(-,-)  is  called  a  correlation  characteristic ,  and  the  functional  equation  (47) 
implies  that  {Xt,t  €  R}  is  a  wide-  sense  Markov  process.  This  means  for  each 
h<t2<  ...  <  tn,n>  1,4*  6  R,  the  (orthogonal)  projection  of  Xtn  on  the  linear 
span  of  Xtl,  ...,Xtn_l  is  the  same  as  the  projection  of  Xtn  on  the  one  dimensional 
span  of  .  (For  a  proof  of  this  classical  fact  see[29],  p.  145.)  It  is  of  some 
interest  to  note  that,  in  the  special  case  of  (46),  if  p(s,t)  —  r(s,  i)/[r(s,  s)r(t,  4)] 
the  correlation  coefficient,  then  also  p  satisfies  the  relation  (47).  If  the  Xt  process 
is  normal  and  (46)  holds,  the  above  noted  projection  becomes  the  conditional 
expectation  and  the  wide-sense  property  becomes  the  usual  (strict-sense)  Markov 
property.  As  an  example,  one  may  consider  r(s,4)  =  exp{-c\s  -  t|),c  >  0. 

A  different  example  of  a  nonstationary  (nonharmonizable)  process  of  multi¬ 
plicity  one  is  the  Brownian  motion.  Here  g(t,X)  =  1  ,F(u)  =  0  if  u  <  0,  =  u,  if 
0  <  u  <  1,  and  =  lifu>l,  AT  =  lin  (42).  If  the  process  is  not  assumed  normal 
(or  Gaussian),  g  s  a  constant,  and  F  is  also  a  constant  outside  of  a  compact  in¬ 
terval,  then  each  nondeterministic  process  of  the  form  (44),  which  has  orthogonal 
increments,  has  multiplicity  one.  Thus  each  of  these  classes  is  large  in  itself.  More 
useful  applications  will  now  be  discussed  in  the  final  two  sections,  (cf.  also  (7).) 

8.  Prediction  and  related  questions  A  linear  least  squares  prediction  of  the 
Xt  process,  by  definition,  is  a  linear  function  of  the  past  and  present  {X,,  u  <  s} 
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which  is  closest  to  Xt,  t  >  a,  so  that  if  Xtt,  is  the  desired  element  in  U,  then 


E( \Xt  -  XtJ 2)  =  inf{E(\Xt  -  7 12)  :  Y  €  *.},  (48) 

where,  as  usual  M,  =  lp{X^.,  u  <  a}  C  Lq(P).  Consequently  Xt =  P,Xt,  with  P, 
as  the  orthogonal  projection  of  U  =  Jp{Xt,t  €  R}  C  Lq(P )  onto 

For  processes  satisfying  the  hypothesis  of  Theorem  7.1,  the  predictor  Xt,»  is 
obtained  immediately.  In  fact,  if  Xt  is  as  above  so  that  it  admits  a  representation 
given  by  (44),  one  has 


=  P.Xt  = 


gk(t,X)Zk(d\), 


(49) 


since  M,  =  (45).  Moreover,  the  minimum  mean  square  error  of  predic¬ 

tion  is  obtained  as 


<7?,.  =  mx,  -  =  E  /'  w*'  >)ian(")-  (so) 

This  in  principle  furnishes  the  desired  solution  of  the  least  squares  linear  prediction 
problem  for  processes  of  the  type  (44).  La  general,  however,  there  is  as  yet  no 
recipe  for  determining  the  multiplicity  of  a  given  continuous  parameter  purely 
nondeterministic  second  order  left  continuous  with  right  limits  process.  But  results 
are  available  if  one  is  willing  to  assume  somewhat  more  on  yjk’s,  generalizing  the 
stationary  case. 

Even  when  the  Xt  process  does  not  satisfy  all  the  conditions  of  Theorem  7.1, 
the  least  squares  prediction  problem  can  be  formulated  and  solved  differently.  To 
understand  this  aspect,  let  {Xt,t  €  T)  C  Lq{P)  be  a  process  and  Ut  =  ap{X,  : 
a  <  t)  as  before.  Suppose  that  M  =  1p{Xt,  t  6  T}  C  L%{P)  is  separable,  which 
holds  if  the  covariance  r(-,  -)  is  continuous  (e.g.,  T  =  Z).  For  each  to  €  T,  the 
best  linear  least  squares  predictor  of  Xto  based  on  the  past  {Xn,u  <  a  <  £0}  is 
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X*0,#  =  P,Xtg((.X9)  and  it  is  the  limit-in-mean  of  linear  combinations  of  X„,  u  <  a. 
On  the  other  hand,  it  is  known  that  a  nonlinear  least  squares  predictor  of  Xt0  is 
given  by  the  conditional  expectation 

r#0l#  =  E(xt0]xu,u<3). 

If  the  process  is  normal  then  one  can  verify  that  Xto,.  =  ^0,..  Thus  for  normal  pro¬ 
cesses,  with  a  continuous  covariance,  both  these  predictors  coincide.  (See  Yaglom 
[34 J,  Chapters  4  and  6  for  a  lucid  discussion  of  these  problems.)  Since  from  a  practi¬ 
cal  point  of  view  it  is  not  feasible  to  have  a  complete  realization  {Xu,  u  <  «}  at  our 
disposal,  it  is  desirable  to  have  some  approximations  to  the  best  predictor.  A  result 
on  this  can  be  described  as  follows.  Let  T  - =  Z  for  simplicity,  and  for  a  <  to  6  Z, 
define  §n  =  ap^X.-i,  ...,X._„}  so  that  lim„  Qn  =  ap{Un<0$„}  =  M,.  If 
Xt0,n  =  Qn(Xto),Q„  being  the  orthogonal  projection  of  U  onto  §n,  then  one  can 
show,  using  the  geometry  of  U ,  that  E{  |X,0,„  -  Xto,*|2)  ^Oasn-^oo.  However, 
the  pointwise  convergence  of  Xt0i„  to  Xt0i,  is  much  more  difficult,  and  in  fact  the 
truth  of  the  general  statement  is  not  known.  For  a  normal  process,  an  affirmative 
answer  can  be  obtained  from  the  following  nonlinear  case. 

Let  Yto,n  =  E(Xto |X„X,_i,  ...,X#_n)  and  Yto, ,  be  as  before.  Then  the  se¬ 
quence  {Yt0,»»,  n  >  1}  is  a  square  integrable  martingale  such  that  supnE{\Yto,n\2)  < 
oo.  Hence  the  general  martingale  convergence  theory  implies  Yto>n  -*•  Yt0i,  both 
in  the  mean  and  with  probability  one,  as  n  — +  oo.  Since  for  normal  processes  both 
the  linear  and  nonlinear  predictors  coincide,  the  remark  at  the  end  of  the  preced¬ 
ing  paragraph  follows.  Thus  predictors  from  finite  but  large  samples  give  good 
(asymptotic)  approximations  for  solutions  Xt0i#  (or  Ytoi#)  and  this  is  important  in 
practical  cases.  However,  the  error  estimation  in  these  problems  received  very  little 
attention  in  the  literature.  In  the  case  of  normal  processes  certain  other  methods 
(e.g.,  the  Kalman  filter  etc.)  giving  an  algorithm  to  compute  the  Xto, ,,-sequence 
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are  available.  But  there  is  no  such  procedure  as  yet  for  the  general  second  order 
processes. 

At  this  point  it  will  be  useful  to  present  a  class  of  nondeterministic  processes, 
belonging  to  a  Karhunen  class,  which  arise  quite  naturally  as  solutions  of  certain 
stochastic  differential  equations.  This  will  also  illustrate  the  remark  made  at  the 
end  of  Section  4. 

In  some  problems  of  physics,  the  motion  Xt  of  a  simple  harmonic  oscillator, 
subject  to  random  disturbances,  can  be  described  by  a  formal  stochastic  differential 
equation  of  the  form  (cf.[3]): 

+  +  “°X<e)  =  Mt)' im  “  (51) 

where  is  the  friction  coefficient  and  u>o  denotes  the  circular  frequency  of  the 
oscillator.  Here  A(t)  is  the  random  fluctuation,  assumed  to  be  the  white  noise— the 
symbolic  (but  really  fictional)  derivative  of  the  Brownian  motion.  In  some  cases  /? 
and  u/o  may  depend  on  time.  To  make  (51)  realistic,  the  symbolic  equation  should 
be  expressed  as: 


dX{t)  +  a1(t)X(t)dt  +  a2{t)X{t)dt  =  dB{t),  (52) 

where  the  B(t)-process  is  Brownian  motion.  Thus  for  each  t  >  0,  B(t)  is  normal 
with  mean  zero  and  variance  a 2t,  denoted  JV(0,cr2t),  and  if  0  <  ti  <  t2  <  tz  then 
B(t3)  —  B(t2 )  and  B[t2)  -  S(ti)  are  independent  normal  random  variables  with 
JV(Q,<72(£3  -  t2)),  N(Q,cr2(t2  -  tx))  respectively.  Also  X(t)  =  is  taken  as  a 
mean  square  derivative.  Then  (52)  and  (51)  can  be  interpreted  in  the  integrated 
form,  i.e.,  by  definition, 


f  f(t)A(t)dt  =  / h  f(t)dB(t ), 

J  a  J  a 


(53) 
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the  right  side  of  (53)  being  a  simple  stochastic  integral  which  is  understood  as 
in  Section  3(since  B  is  also  orthogonally  scattered).  Here  /  is  a  nonstochastic 
function.  The  integration  theory,  if  /  is  stochastic  needs  a  more  subtle  treatment 
and  the  f?(t)-process  can  also  be  replaced  by  a  “semi-martingale.”  (See,  e.g.,  [26], 
Chapter  IV  and  V  for  details.)  The  point  is  that  the  following  statements  have 
a  satisfactory  and  rigorous  justification.  With  Brownian  motion  one  can  assert 
more,  and,  in  fact  regarding  the  solution  process  of  (52),  the  following  is  true. 

Theorem  8.1  Let  J  =  [ao,6oj  C  R+  be  a  bounded  interval ,  and  {Bt,t  €  J} 
be  the  Brownian  motion.  If  a,-(*),  i  =  1,2  are  real  (Lebesgue)  integrable  functions 
on  J  such  that  equation  (52)  is  valid,  then  there  exists  a  unique  solution  process 
{Xt,t  €  J}  satisfying  the  initial  conditions  Xao  =  C i,Xao  =  C2  where  CltC2  are 
constants.  In  fact,  the  solution  is  defined  by. 

Xt=  f  G{t,u)dB{u)+C1Vl(t)+C2V2{t),  t€J,  (54) 

J  *0 

where  Vi(-),  t  =  1,2  are  the  unique  solutions  of  the  accompanying  homogeneous 
differential  equation: 


~ J ^  +  "2  (*)/(*)  =  0  (55) 

with  the  initial  conditions  /(a0)  =  l,/(a0)  =  0,  and  f(a0)  =  0 ,/(a0)  =  1  respec¬ 
tively.  In  (54),  G  :  J  x  J  —*  C  is  the  Green  function.  This  is  a  continuous  function 
such  that  Off-  is  continuous  in  (t,s)  on  a<j  <  t  <  s  <  b0,  and  has  a  jump  on  the 
diagonal,  i.e., 


+  0,  s)  -  —  (s  -  0 ,s)  =  1.  (56) 

Moreover,  the  Xt  process  given  by  (54)  is  of  Karhunen  class  and  is  purely  nondeter- 
ministic  (since  the  spectral  function  of  B  is  the  Lebesgue  measure).  Its  covariance 
r  is  given  by 
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(57) 


r(s,t)  =  /  G(s, \)G(t,  X)dX,  a,t  e  J , 

J  ao 

and  the  process  has  multiplicity  one. 

This  result  shows  that  the  processes  appearing  as  solutions  of  the  (linear) 
stochastic  differential  equations  have  interesting  special  properties.  Further  one 
can  show  that  the  vector  process  G  J}  is  a  (vector)  Markov  normal 

process  almost  all  of  whose  sample  paths  are  continuous.  (For  details  of  these 
assertions,  see  [25],  Sec.  4.)  Related  results  for  the  ntfc-order  case  with  continuously 
(n-1)  times  differentiable  coefficients  a,  (  ),  and  initial  conditions  C,  =  0  have  been 
analyzed  by  C.  L.  Dolph  and  M.  A.  Woodbury  [8].  The  work  exemplifies  the 
importance  of  nondeterministic  processes  of  multiplicity  one  in  applications  coming 
from  both  the  physical  sciences  and  communication  theory.  Let  us  now  turn  to 
another  type  of  application. 

The  general  filtering  problem  can  be  presented,  following  Bochner  [2],  as  fol¬ 
lows.  Let  Xt  be  the  set  of  all  second  order  process  X  =  {Xt,  t  6  T},  with  zero 
means.  Let  A  be  a  linear  operator  on  the  linear  space  Xt-  Suppose  X,Y  €  Xt 
and  that  X  €  domain(A),  and 


A X=Y  or  (AX)t,  =  Yt,  t<=T.  (58) 

As  usual  T  =  Z,  or  =  R.  Typically  the  Y-process  is  the  output  and  the  X  process, 
the  input  and  A  is  termed  a  (linear)  filter.  The  problem  here  is  to  find  conditions  on 
A  such  that  if  the  output  is  known  one  can  recover  the  input  process.  As  examples, 


{hX)t  —  aiXt-i,  t  €  Z,  o,-  €  R, 

«=i 


Xt-*f(u)du, 


t  €  R,  suitable  /. 


The  first  one  is  called  a  difference  or  a  polynomial  or  a  moving  average  filter,  and 
the  second  one,  an  integral  filter,  one  can  also  have  a  difference-differential  or  an 
integro-differential  filter  and  the  like.  If  both  X ,  Y  are  stationary  processes  and  A 
is  a  polynomial  or  an  integral  filter,  then  precise  conditions  for  recovering  the  input 
were  first  obtained  by  Nagabhushanam  [19].  His  results  were  extended  by  Kelsh 
[15]  if  X,  Y  are  strongly  harmonizable,  and  (using  necessarily  different  methods) 
a  further  extension  to  the  weakly  harmonizable  case  by  Chang.  For  an  exposition 
of  these  results  with  a  numerical  illustration,  one  may  refer  to  [4]. 

The  above  described  filtering  problem  changes  its  character  if  only  a  finite 
segment  of  the  observations  on  {Xt,  t  G  T}  is  available.  Assuming  a  knowledge 
of  the  covariance  structure  (from  prior  information)  of  the  process,  how  can  one 
estimate  an  element  of  the  process  (prediction  or  interpolation)  which  is  not  part 
of  the  observed  ones?  While  a  precise  set  of  conditions  is  difficult  to  obtain,  good 
sufficient  conditions  can  be  given  for  its  solution.  This  point  will  be  discussed  by 
a  specialization  and  adjustment  of  the  work  from  [8]. 

Let  {Xt,0  <  t  <  1}  C  L2(P)  be  an  observed  process  which  is  known  to  have 
a  linear  time  trend  and  a  random  disturbance.  Thus  the  model  is  given  as: 

Xt  =  a  +  6t  + Yi,  0  <  t  <  1  (59) 

where  a,  b  are  real  but  unknown  constants  and  {Yt,0  <  t  <  1}  is  a  noise  process 
which  is  assumed  to  be  stationary  with  mean  zero  and  covariance  r,  given  by 
r(s,t)  =  exp(—  0)s  — 1]),/9  >  0.  The  problem  is  to  find  an  unbiased  linear  estimator 
of  Xto,t0  >  1,  based  on  the  output  {Xt,0  <  t  <  1},  using  the  least  squares 
criterion.  This  can  be  made  more  explicit  as  follows:  it  is  desired  to  find  a  weight 
function  u>(-)  on  0  <  t  <  1  which  is  of  bounded  variation  such  that  if 

Xt„(u>)  =  [  Xtdw{t).  (60) 

Jo 
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is  the  linear  estimator  then  E{Xto(w))  =  a  +  bto,  a,  6  6  R  and  S(|Xe0  — Xto(t£>)|2) 
is  a  minimum.  Since  r  and  hence  0  are  known,  let  us  take  0  =  1  for  this  illustration. 
Then  subject  to  the  unbiasedness  constraint,  one  can  minimize  the  mean  square 
error  using  the  variational  calculus  (or  Lagrange  multipliers)  and  show  that  there 
exists  a  w<  ht  w(-)  having  a  density  wl.  Thus  after  a  calculation,  one  finds  w'  and 
Xt(w)  to  be: 


2 w\t)  =  —[56  -  36 to  +  36*(2*0  -  1)] 
0/ 


i  rl 

Xto{w)  =  rr {  /  Xt[2S  -  18*o  +  18*(2*o  -  l)]dt 
Jo 

-2X1(4~27*o)~2X0(27*o-23)}.  (62) 

The  mean  square  error  for  this  problem  is  then 

<r2  =  E(\Xt0  -  Xt0(«;)|2)  =  [72^  -  72t0  +  56]/ 57.  (63) 

The  actual  details  of  computations  for  (61)-(63)  involve  solving  an  integral  equation 
and  thus  are  not  entirely  simple.  It  is  however  interesting  to  remark  that,  in  this 
calculation,  one  need  not  find  estimators  of  a,  b  separately  from  the  data,  and  the 
variational  calculus  enables  a  direct  solution  as  indicated. 

The  problem  of  minimum  variance  unbiased  (linear)  estimator  of  a,  b  is  also 
important.  For  instance,  if  b  =  0  in  (59),  then  an  estimator  of  the  unknown 
parameter  a  (i.e.  estimating  the  mean  of  the  X  process  without  trend)  can  be 
obtained  by  a  similar  method.  The  estimator  a  of  the  form 

d(to)  =  [  Xtdtv(t),  E[ d(w))  -  a,  a  €  R, 

Jo 
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has  been  considered  by  Grenander  [10],  and  the  result  is: 

CL  =  h[l  Xtdt  +  Xi+X o].  (64) 

3  J  o 

Other  estimation  methods  and  their  properties  are  discussed  for  stationary  error 
processes  in  [11]. 

9.  Some  inference  problems  with  normal  processes.  In  this  final  section  some 
special  inference  questions  when  the  processes  are  normal  are  briefly  discussed  to 
supplement  the  preceding  work. 

Recall  that  a  normal  process  {Xt,t  €  T}  is  a  collection  of  random  variables 
such  that  each  finite  subset  has  a  joint  normal  distribution.  Now  if  (0,  E,  P,),  i  = 
1,2  are  a  pair  of  probability  spaces  with  a  common  base  space  G,  then  Pj  and 
P2  are  said  to  be  mutually  singular  or  perpendicular  (written  Pi  ±  P2)  if  there  is 
an  event  Aq  €  E  such  that  Pi(Ao)  =  0  and  P2(Ao)  =  1,  and  mutually  absolutely 
continuous  or  equivalent  (written  Pi  ~  P2)  if  both  Pi  and  P2  vanish  on  the  same 
class  of  sets  from  E.  For  instance,  if  G  =  R,  E  =  the  Borel  <r-algebra,  Pi  =  normal 
and  P2  =  Cauchy,  then  Pi  ~  Q.  On  the  other  hand  if  Pi  is  normal  and  P2  is 
Poisson,  then  Pi  JL  P2.  However,  if  G  =  RT,  E  =  the  cylinder  a-  algebra,  then 
Xt  :  G  — ►  R  is  defined  as  Xt(w)  =  w(t),  i.e.,  the  coordinate  function,  and  the 
problem  of  determining  as  to  when  Pi  ~  P2,  or  Pi  J.  P2,  or  neither,  is  not  simple. 
In  the  case  that  both  Pi,P2  are  normal  probability  measures  on  G  =  Rr,  only  the 
main  dichotomy  that  Pi  ~  P2  or  Pj  _L  P2  can  occur.  This  was  first  established 
independently  by  J.  Feldman  and  J.  Hdjek  in  1958  and  later  elementary  proofs 
of  this  theorem  were  presented  by  L.A.  Shepp  and  others.  A  simplified  but  still 
nontrivial  proof  of  this  result  with  complete  details  is  given  in  ([27],  pp.  212-217). 
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The  statistical  problem  therefore  is  to  decide,  on  the  basis  of  a  realization, 
which  one  of  Pi ,  P2  is  the  correct  probability  governing  the  process.  In  the  singular 
case,  this  is  somewhat  easier,  but  in  case  Pi  ~  P2,  the  problem  is  not  simple.  A 
number  of  cases  have  been  discussed  in  [10]  before  the  dichotomy  result  is  known. 
The  simplest  usable  condition  in  the  general  case  is  the  following. 

Let  Pi  have  the  mean  and  covariance  functions  (mi,r,-),  written  P(m,-,  r,),t  = 
1,2.  Then  Pi  ~  P2  iff  one  has  P(0,  fi)  ~  P(0,r2)  and  P(mi,ri)  ~  P(m2,rx). 
Thus  P(mi,ri)  ~  P(m2,r2)  iff  P^i,^)  ~  P(m2, »*i)  ~  P(nt2,r2).  Some  appli¬ 
cations  with  likelihood  ratios  appear  in  [25].  This  equivalence  criterion  will  now 
be  illustrated  on  a  purely  nondeterministic  normal  process  of  multiplicity  one. 

If  { Xt,t  €  T}  is  a  normal  process  with  mean  zero  and  covariance  r  let  Zt  = 
m(t)  +  Xt  where  m  :  T  — ♦  R  is  a  measurable  nonstochastic  function,  so  that  the 
Zt*process  has  mean  function  m  and  covariance  r  and  is  also  normal.  Let  P  and 
Pm  be  the  corresponding  probabilities  governing  them.  The  mean  m(  )  is  called 
admissible  if  P  ~  Pm.  The  set  Mp  of  all  admissible  means  is  an  interesting  space 
in  its  own  right.  In  fact,  it  is  a  linear  space,  carries  an  inner  product  and  with 
it  Mp  becomes  a  Hilbert  space  attached  to  the  given  normal  process.  (For  an 
analysis  of  MP,  and  the  following,  see  [24].)  One  shows  that  m  €  MP  iff  there  is 
a  unique  Y  e  U  =  sp{Xt,  t  €  T}  C  Lq(P)  such  that 

m(t)  =  E{YXt),  ter  (65) 

and  then  the  likelihood  ratio  is  given  by 

^  =  *zp{Y  -  ^E(\Y\2)}.  (66) 

Using  now  an  abstract  generalization  of  the  classical  Neyman-Pearson  lemma  due 
to  Grenander  ([10],  p.210),  one  can  test  the  hypothesis  H0  :  m  =  0,  vs,  Hi  :  m(t)  £ 
0.  The  critical  region  for  this  problem  can  be  shown  to  be 
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Ak  =  {w  6  n  :  Y(u)  <  Jfe},  (67) 

where  Jc  is  chosen  so  that  P(A* )  =  a,  the  prescribed  size  of  the  test  (e.g.,  a  =  0.05 
or  0.01).  This  general  result  was  first  obtained  by  Pitcher  [23].  In  the  case  of 
nondeterministic  processes  of  multiplicity  one,  the  conditions  on  admissible  means 
can  be  simplified  much  further.  This  may  be  stated  following  Cramer  [7],  as  follows. 

Let  T  =  [a,  6]  and  Xt  be  purely  nondeterministic  so  that  by  (44)  with  N  —  1, 
one  has 


Xt=  f  g(t,\)Z(d\),  teT,  (68) 

Ja 

and  that  U  =  «p{Xt  €  T}  =  7p{Z(A)  :  A  C  T,Borel}.  But  m  €  Mp  iff  there 
exists  a  Y  €  U  such  that  (65)  holds.  In  this  special  case  therefore,  Y  admits  a 
representation  as 


Y 


L 


=  /  h(X)Z(i\), 


(69) 


for  some  h  €  L2([a,b],F)  where  P(A)  =  E(\Z(A)\2).  Suppose  that  the  deriva¬ 
tive  F'  exists  outside  a  set  of  Lebesgue  measure  zero.  Since  Z(-)  has  orthogonal 
increments,  (65),  (68)  and  (69)  imply 


m{t)=  f  h{\)g{t,\)F'{\)d\,  teT  =  [a,b}.  (70) 

J  a 

This  is  the  simplification  noted  above.  If  is  assumed  to  exist,  then  (70)  implies 
that  the  derivative  m'(i)  of  m(t)  also  exists.  In  particular  if  the  Xt  is  the  Brownian 
motion  so  that  g  =  1  and  F'  =  1,  one  gets  rn'(t)  =  h(t),  (a.e.)  and  h  6  L2([a,  6],  dt) 
in  order  that  Pm  P, 

There  is  a  corresponding  result,  when  P\  ~  P2 ,  P,  are  normal,  but  have  dif¬ 
ferent  covariances.  However,  this  is  more  involved.  A  discussion  of  this  case  from 
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different  points  of  view  occurs  in  the  works  [35],  [33],  [7]  and  [25].  (See  also  the 
extensive  bibliography  in  these  papers.)  There  is  a  great  deal  of  specialized  anal¬ 
ysis  for  normal  process  in  both  the  stationary  and  general  cases.  It  is  thus  clear 
how  various  types  of  techniques  can  be  profitably  employed  to  several  classes  of 
nonstationary  processes  of  second  order.  Many  realistic  problems  raised  by  the 
above  work  are  of  interest  for  future  investigations. 

Acknowledgement.  This  work  is  prepared  with  a  partial  support  of  an  ONR 
Contract. 
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